Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added image-to-image task for ORT Pipeline #2031

Conversation

h3110Fr13nd
Copy link
Contributor

@h3110Fr13nd h3110Fr13nd commented Sep 19, 2024

What does this PR do?

Fixes #2030
Image To Image task consist lot more than Diffusion and there are transformer based model such as Swin.
This PR adds the support for pielining image-to-image task with ORTModelForImage2Image.
e.g.

from transformers import AutoImageProcessor
from optimum.pipelines import pipeline
from optimum.onnxruntime import ORTModelForImageToImage
from PIL import Image

model = ORTModelForImageToImage.from_pretrained("swin2sr-2x-onnx", use_io_binding=True)
processor = AutoImageProcessor.from_pretrained("caidas/swin2SR-classical-sr-x2-64")
onnx = pipeline("image-to-image", model=model, feature_extractor=processor, device="cuda")
image_path = "image.png"
image = Image.open(image_path)
output = onnx(image)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

Who can review?

@fxmarty, @echarlaix, @JingyaHuang, @michaelbenayoun

@IlyasMoutawwakil
Copy link
Member

awesome ! don't hesitate to ping me for review when it's ready !

@h3110Fr13nd
Copy link
Contributor Author

awesome ! don't hesitate to ping me for review when it's ready !

Sure. Writing some tests. Will update you soon.

@h3110Fr13nd h3110Fr13nd marked this pull request as ready for review September 23, 2024 22:15
@h3110Fr13nd
Copy link
Contributor Author

h3110Fr13nd commented Sep 23, 2024

Suggest Changes if Any.
@IlyasMoutawwakil, @fxmarty, @JingyaHuang, @michaelbenayoun

Copy link
Member

@IlyasMoutawwakil IlyasMoutawwakil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, I left a few comments and suggestions, mostly to follow the same testing pipeline we have for other ORTModels:

  • base the TestCase on a simpler task (I think u based on generative tasks)
  • use setup to export/use an exported model
  • use tiny models for faster testing

Copy link
Member

@IlyasMoutawwakil IlyasMoutawwakil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !
did you test io binding locally ?

@h3110Fr13nd
Copy link
Contributor Author

LGTM ! did you test io binding locally ?

Yes.

@h3110Fr13nd
Copy link
Contributor Author

Should I add a test for that?

@IlyasMoutawwakil
Copy link
Member

we won't be able to test it until #2024 is re-enabled. will merge this now.

@IlyasMoutawwakil IlyasMoutawwakil merged commit fd638d2 into huggingface:main Sep 26, 2024
41 of 45 checks passed
@h3110Fr13nd h3110Fr13nd deleted the feat/added-ort-model-for-image-to-image-task branch September 26, 2024 11:49
@h3110Fr13nd h3110Fr13nd restored the feat/added-ort-model-for-image-to-image-task branch September 26, 2024 11:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

OrtModelForImage2Image - For Onnx runtime on image-to-image tasks.
2 participants